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IMPERATIVES: 

•  Continued  Cost  and  Schedule  Improvements  in  DOD’s  MDAP  Portfolio 

•  Increased  Focus  on  Truly  Understanding  Requirements 

•  Greater  Efficiencies  by  Continuing  Improvements  to  the  Acquisition  Process 
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DEFENSE  ACQUISITIONS: 
Assessments  of  Selected  Weapon  Programs 

(GAO-1 6-329SP) 


Selected  Findings  from  GAO’s  Analysis  of  DOD’s 
Portfolio  of  Major  Defense  Acquisition  Programs 


For  more  information,  contact  J.  Kristopher  Keener  202-512-9310 


GAO 


Objectives 


This  is  GAO's  14th  annual  assessment  of  DOD  weapon  system  acquisitions,  an  area  that 
has  been  on  GAO's  high-risk  list  for  24  years.  The  report  assessed  selected  weapon 
programs  in  DOD’s  fiscal  year  2015  major  defense  acquisition  portfolio. 


Our  objectives  were  to  determine: 

1 )  how  DOD’s  portfolio  of  79  major  defense  acquisitions  performed  over  time  in 
terms  of  both  cost  and  schedule; 

2)  to  what  extent  43  selected  weapon  acquisition  programs  demonstrated  key 
product  knowledge  at  specific  milestones  in  the  acquisition  process; 

3)  to  what  extent  43  current  and  12  future  major  acquisition  programs  implemented 
key  reform  initiatives  and  addressed  concurrent  acquisition  activities. 
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Observations  on  the  cost  and  schedule  performance 
of  DOD’s  major  acquisition  programs 


Changes  across  portfolios 

1.  When  compared  to  the  2014  portfolio,  the  cost  of  the  2015  portfolio  decreased  by  nearly  $15  billion  from 
$1 ,455  to  $1 ,440  billion  and  the  number  of  programs  in  the  portfolio  increased  by  one  to  79.* 

2.  The  total  amount  of  funding  required  for  the  portfolio  has  been  decreasing  since  2010.  The  future  funding 
needed  to  complete  the  portfolio — $603  billion — is  at  its  lowest  point  in  over  a  decade.  Of  this  future  funding, 
$571  billion  is  for  procurement  and  $32  billion  is  for  development.  The  low  amount  of  future  development 
funding  required  is  likely  due  to  newer  programs  with  less  risk  entering  system  development. 

Changes  in  the  2015  portfolio 

3.  When  analyzing  the  change  to  cost  and  schedule  estimates  over  the  past  year  for  the  79  programs  in  the 
2015  portfolio,  the  estimated  total  acquisition  cost  decreased  by  $2.5  billion  and  the  average  schedule  delay 
in  achieving  initial  capability  increased  by  2.4  months.  When  assessed  against  first  full  estimates,  total  costs 
have  increased  by  $469  billion,  over  48  percent,  and  the  average  delay  in  delivering  initial  capabilities  has 
increased  to  almost  30  months. 

Factors  that  explain  changes  in  the  2015  portfolio 

4.  Older  programs  carry  a  majority  of  the  total  cost  and  cost  growth  since  first  full  estimates.  Of  the  79 
programs  in  the  2015  portfolio,  40  were  also  in  the  2005  portfolio  representing  80  percent  of  the 
portfolio’s  total  acquisition  cost. 

5.  While  the  total  cost  of  the  2015  portfolio  decreased,  42  of  the  79  programs  increased  in  cost  over  the  past 
year.  Cost  estimate  decreases  on  37  programs  resulted  in  the  overall  net  cost  decrease. 


*We  exclude  the  Ballistic  Missile  Defense  System  as  its  reported  baselines  are  not  comparable  with  other  programs. 
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Observations  on  the  cost  and  schedule  performance 
of  DOD’s  major  acquisition  programs 

Factors  that  explain  changes  in  the  2015  portfolio  and  other  observations 

6.  38  programs  gained  buying  power  during  the  past  year  resulting  in  a  net  gain  of  $10.7  billion.  Six 
programs  have  demonstrated  buying  power  gains  or  losses  annually  over  the  past  5  years. 

7.  Schedule  delays  over  the  past  year  in  11  of  the  79  programs  contributed  to  the  portfolio’s  overall  delay  of 
2.4  months.  The  Airborne  Maritime/Fixed  Station  radio  system  experienced  the  largest  delay  at  more  than  8 
years,  which  disproportionately  affected  the  portfolio. 

8.  Over  the  past  year,  16  programs  reported  development  cost  growth  while  in  production.  This 
represents  concurrency,  which  can  be  caused  by  many  factors,  and  is  a  contributor  to  cost  growth. 

9.  As  measured  against  the  metrics  discussed  with  the  Office  of  Management  and  Budget  and  DOD, 
more  programs  meet  each  metric  for  cost  change  than  our  last  assessment. 

10.  The  total  cost  of  the  portfolio  estimated  for  Army  programs  is  the  smallest  of  the  services  and  has 
been  decreasing  since  2007.  The  Navy  and  Air  Force’s  share  of  portfolio  costs  have  been  increasing 
since  2007  and  2012  respectively.  Navy  programs  account  for  almost  55  percent  of  the  2015 
portfolio’s  total  estimated  acquisition  cost. 

1 1 .  The  equity  prices  of  contractors  delivering  the  ten  costliest  programs  performed  well  relative  to 
broad-based  market  indices,  indicating  that  investors  expect  them  to  remain  profitable  in  the  future. 
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OBJ.  1  Obs.  2:  Portfolio  Comparisons 


GAO 

The  decrease  in  cost  from  the  2014  portfolio  to  the  2015 
portfolio  continues  a  trend  identified  in  previous  assessments 

DOD  Portfolio  Cost  and  Size,  2005-2015 

Fiscal  year  2016  dollars  (in  billions)  Number  of  programs 


Total  acquisition  cost 

-  Number  of  programs 

Source:  GAO  analysis  of  DOD  data.  |  GAO-1 6-329SR 

Note:  The  2009  portfolio  is  excluded  because  there  were  no  annual  Selected  Acquisition  Reports  (SAR)  released  for  the  December 
2008  submission  date. 

•  DOD’s  portfolio  has  been  steadily  decreasing  in  terms  of  cost  since  2010. 

•  Over  half,  or  40  of  the  79  programs  in  the  2015  portfolio,  were  also  in  the  2005  portfolio  and 
represent  80  percent  of  the  portfolio’s  current  total  acquisition  cost  or  over  $1.1  of  the  $1 .4  trillion. 
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OBJ.  1  Obs.  2:  Portfolio  Comparisons 


The  total  amount  of  funding  required  for  the  portfolio  has  been  decreasing  since 
2010,  the  amount  needed  to  complete  the  portfolio — $603  billion — is  at  its  lowest 
point  in  over  a  decade 

Future  Development  and  Procurement  Funding  in  Comparison  to  Invested  Funding  by  Year,  2005-2015 

Fiscal  year  2016  dollars  (in  billions) 
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■  Total  development  and  procurement  funding  invested  in  the  portfolio  ■  Development  funding  needed  to  complete  the  portfolio 
Procurement  funding  needed  to  complete  the  portfolio 
Source:  GAO  analysis  of  DOD  data. 

Note:  The  2009  portfolio  is  excluded  because  there  were  no  annual  SARs  released  for  the  December  2008  submission  date. 

•  Overall,  the  amount  of  total  funding  needed  to  complete  the  portfolio  has  been  decreasing  since 
2006  and  is  currently  $603  billion  or  42  percent  of  the  portfolio’s  total  cost. 

•  Of  note  is  the  decrease  in  the  amount  of  development  funding  needed  to  complete  the  portfolio. 
Only  $32  billion  in  development  remains  to  be  appropriated  down  from  almost  $128  billion  in  2005. 
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OBJ.  1  Obs.  3:  Changes  over  the  Past  Year  in  the  2015  Portfolio 


Over  the  past  year,  the  estimated  cost  of  the  2015  portfolio  decreased  by  $2.5 
billion,  the  average  delay  in  achieving  initial  capability  increased  by  2.4  months 


Cost  and  Schedule  Change  for  the  79  Programs  in  the  2015  Portfolio 

Fiscal  year  2016  dollars  (in  billions) 

Estimated 

Estimated 

Estimated 

Percentage 

portfolio  cost  in 

portfolio  cost  in 

portfolio  change 

change  since 

2014 

2015 

since  2014 

2014 

Total  estimated  research  and  development  cost 

$285.9 

$289.0 

$3.1 

1.1% 

Total  estimated  procurement  cost 

1,143.5 

1,137.6 

-6.0 

-0.5 

Total  estimated  acquisition  cost 

1,442.0 

1,439.6 

-2.5 

-0.2 

Average  delay  from  first  full  estimate  in  delivering  initial 

27  months 

29.5  months 

2.4  months 

1.3 

capabilities 

additional  delay 

Source:  GAO  analysis  of  DOD  data. 

Note:  In  addition  to  research  and  development  and  procurement  costs,  total  acquisition  cost  includes  acquisition  related  operation  and  maintenance  and 
system-specific  military  construction  costs.  Some  numbers  may  not  sum  due  to  rounding. 


•  The  total  procurement  cost  decreased,  similar  to  our  last  assessment. 

•  The  2.4  month  average  delay  in  delivering  operational  capability  is  a  month  greater  than 
our  last  assessment  and  continues  a  trend  we  have  seen  for  the  past  decade. 

•  When  measured  from  first  full  estimates  the  total  estimated  cost  of  the  portfolio  has 
grown  by  over  48  percent,  or  nearly  $469  billion. 

•  Eighty  percent  of  cost  growth  since  first  full  estimates  occurred  five  or  more  years  ago. 
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OBJ.  1  Obs.  5:  Distribution  of  1-year  Total  Acquisition  Cost  Change 


GAO 

Although  the  total  cost  for  the  2015  portfolio  decreased,  42  of  79  programs  increased  in  cost 
over  the  past  year,  the  cost  decreases  on  the  remaining  37  programs  resulted  in  the  net  cost 
decrease 


Distribution  of  the  1-year  Change  in  Total  Acquisition  Cost  for  the  2015  Portfolio 


significant  cost  changes,  i.e.  no  “outliers”. 

•  Cost  changes  resulted  from  a  variety  of  program  changes,  including  changes  to  program  quantities. 
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OBJ.  1  Obs.  6:  Buying  Power  Analysis 


GAO 

Accounting  for  quantity  changes,  38  programs  in  the  portfolio  gained  buying 
power  during  the  past  year  resulting  in  a  net  buying  power  gain  of  $10.7  billion 


Procurement  Cost  Changes  in  the  2015  Portfolio  due  to  Quantity  Changes  and  Other  Factors  Fiscal  year  2016  dollars  (in  billions) 


Number  of 

programs 

GAO 
Calculated 
Cost  Change 

Actual 
Procurement 
Cost  Change 

Cost  change  not 
attributable  to 
quantity 

Increased  buying  power 

38 

$10.6 

-$5.4 

-$16.0 

Procurement  cost  decreased  with  no  quantity  change 

26 

$0.0 

-$10.6 

-$10.6 

Quantity  increased  with  less  cost  increase  than  anticipated 

11 

$10.6 

$5.9 

-$4.7 

Quantity  decreased  with  more  cost  decrease  than  anticipated 

1 

-$.06 

-$0.7 

-$0.7 

Decreased  buying  power 

35 

-$4.2 

$1.1 

$5.3 

Procurement  cost  increased  with  no  quantity  change 

25 

$0.0 

$2.0 

$2.0 

Quantity  increased  with  more  cost  increase  than  anticipated 

6 

$4.9 

$5.5 

$0.6 

Quantity  decreased  with  less  cost  decrease  than  anticipated 

4 

-$9.1 

-$6.3 

$2.8 

No  change  in  buying  power 

5 

$0.0 

$0.0 

$0.0 

Program  eliminated  procurement 

1 

-$1.7 

-$1.7 

$0.0 

Portfolio  totals 

79 

$4.7 

-$6.0 

-$10.7 

Source:  GAO  analysis  of  DOD  data. 

•  Our  analysis  shows  that  38  programs  increased  their  buying  power  in  the  past  year  and  reduced  procurement  costs 
by  a  total  of  $5.4  billion.  This  total  is  the  net  amount  of  cost  change  given  the  $10.6  billion  in  increases  due  to  quantity 
increases  and  the  $16  billion  in  decreases  due  to  other  program  efficiencies. 

•  These  buying  power  gains  outweighed  losses  resulting  in  a  net  buying  power  gain  of  $1 0.7  billion. 
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OBJ.  1  Obs.  10:  Services’  Share  of  Portfolio  Cost 


GAO 

The  total  cost  of  the  portfolio  estimated  for  Army  programs  is  the  smallest  of  the 
services  and  has  been  decreasing  since  2007.  The  Navy  and  Air  Force’s  share  of 
portfolio  costs  have  been  increasing. 


Services’  Percentage  of  Total  Portfolio,  2005-2015 

Percentage  of  Total  Acquisition  Cost  by  component 

0.7  0.7  0.8  4.4  0.8  1.0  1.1  1.1  1.4  1.4 


o - 

2005  2006  2007  2008  //  2010  2011  2012  2013  2014  2015 

Fiscal  year 


Source:  GAO  analysis  of  DOD  data.  |  GAO-16-329SP 


Office  of  the  Secretary  of  Defense 
Army 

•  Spending  on  Navy  programs  now 
accounts  for  55  percent  of  the 
portfolio  or  $54.6  billion. 

m  Air  Force 

•  As  the  share  of  cost  for  Navy 
programs  increased,  that  for  the 
Army  has  decreased,  it  is  now  at  10 
percent  of  the  portfolio. 


^  Navy  •  Some  Navy  programs,  shipbuilding 
in  particular,  have  experienced 
significant  cost  growth. 


•  In  a  constrained  funding 
environment,  unforeseen  cost 
increases  limit  investment  choices. 
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OBJ.  1  Obs.  11:  Industry  Effects 


GAO 

The  equity  prices  of  the  contractors  delivering  the  ten  costliest  programs  have 
performed  well  relative  to  broad-based  market  indices,  indicating  that  investors 
expect  these  firms  to  remain  profitable  well  into  the  future 


•  Five  publicly-traded  defense  contractors  are  developing  and 
delivering  the  ten  largest  DOD  programs  in  the  2015  portfolio. 

•  These  10  programs  account  for  $866  billion  in  acquisition  costs, 
over  60  percent  of  the  total  portfolio. 

•  The  equity  prices — a  stock  price-based  indicator  of  investor 
expectations  of  future  earnings  over  many  years — for  these 
contractors  over  the  past  decade  has  increased  at  a  rate  that 
outperformed  broad-based  indices  for  markets  as  a  whole,  as  well 
as  narrower  indices  covering  similar  sectors  of  the  economy. 

•  This  indicates  that  investors  expect  the  performance  of  these 
companies  to  be  particularly  strong  for  some  time  to  come. 

•  Strong  equity  performance  among  these  contractors  could  be 
driven  by  a  number  of  factors  ana  is  not  necessarily  related  to 
the  financial  condition  of  major  acquisition  programs. 
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OBJ.  2  Obs.  1  -  Development  Start  for  Future  Programs 


Of  the  12  future  programs  we  assessed,  5  identified  critical  technologies  and  their 
anticipated  maturity  levels 


Program 

Development  start 

Projected  to  demonstrate  all 
critical  technologies  in  an 
operational  environment 

Projected  to  complete 
all  systems  engineering 
reviews 

Plan  to  constrain 
system 
development 

T-AO(X) 

06/2016 

• 

O 

• 

IFPC  Inc.  2 

06/2016 

• 

• 

F-15  EPAWSS 

09/2016 

O 

• 

• 

OR 

09/2016 

0 

O 

PAR 

03/2017 

0 

O 

P-8A  Inc.  3 

06/2017 

0 

0 

UCLASS 

06/2017 

• 

_ 

0 

JSTARS  Recap 

09/2017 

_ 

• 

• 

ITEP 

06/2018 

0 

0 

0 

LX(R) 

09/2018 

0 

0 

APT 

12/2017 

0 

• 

WSF 

12/2018 

0 

0 

0 

•  Implementation  planned 

0  No  implementation  planned 
—  Practice  to  be  determined 

Critical  technologies  have  not  been  identified  per  the  program  office  response 

•  The  best  point  of  leverage  to  ensure  a  program’s  success  is  at  the  start  of  a  new  program.  At  this  point,  decision  makers  can  ensure 
that  new  programs  implement  best  practices  and  exhibit  desirable  principles  before  they  are  approved  and  funded. 

•  Three  of  the  12  programs  plan  to  conduct  all  of  the  recommended  systems  engineering  reviews  before  development  start,  including  a 
system  functional  review,  a  system  requirements  review,  and  a  preliminary  design  review. 
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OBJ.  2  Obs.  1  -  Development  Start 


The  seven  programs  that  started  system  development,  or  planned  to,  during  our 
assessment  period  have  knowledge  deficits  which  introduce  risk  and  could  lead  to 
cost  and  schedule  growth 


Knowledge-based  practices 
at  system  development  start 


-ik 

o 


0} 


& 


S' 


& 


2 


J? 


Other  36 
programs 

/o/- 


Demonstrate  all  critical  technologies  in  a 
relevant  environment 

• 

• 

• 

• 

• 

• 

o 

20 

8 

8 

Demonstrate  all  critical  technologies  in  an 
operational  environment 

o 

o 

o 

o 

o 

o 

o 

3 

23 

10 

Complete  system  functional  review  and 
system  requirements  review  before 
system  development  start 

o 

• 

• 

• 

• 

• 

• 

10 

25 

1 

Completed  preliminary  design  review 
before  system  development  start 

o 

• 

• 

o 

• 

• 

• 

13 

22 

1 

Constrain  system  development  phase  to 

6  years  or  less 

• 

• 

• 

• 

— 

• 

• 

22 

6 

8 

•  OASuW  Inc.  1  plans  to 
have  immature 
technologies  at  the 
start  of  system 
development. 

•  None  of  the  programs 
demonstrated  critical 
technologies  in  an 
operational 
environment. 


o 


Practice  implemented 
Practice  not  implemented 

Practice  not  applicable  or  information  not  available  per  the  program  office  response 


ACV  -  Amphibious  Combat  Vehicle 

B-2  DMS-M  -  B-2  Defensive  Management  System  Modernization 

CIRCM  -  Common  Infrared  Countermeasures 

JAGM  -  Joint  Air-to-Ground  Missile 

MGUE  -  Military  GPS  User  Equipment  Increment  1 

NGJ  Inc.  1  -  Next  Generation  Jammer  Increment  1 

OASuW  Inc.  1  -  Offensive  Anti-Surface  Warfare  Increment  1 


Source:  GAO  analysis  of  DOD  data  |  GAO-16-329SP 


•  Five  programs 
conducted  all  systems 
engineering  reviews, 
including  a  preliminary 
design  review,  before 
entering  system 
development. 
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OBJ.  2  Obs.  2  -  Critical  Design  Review 


Of  the  five  programs  that  held,  or  are  planning  to  hold,  a  critical  design  review 
during  our  assessment  period,  two  met  all  of  the  best  practices 


Knowledge-based  practices 
at  critical  design  review 


Demonstrate  all  critical  technologies  in  an 
operational  environment 

o 

o 

• 

• 

• 

3 

17 

6 

Release  at  least  90  percent  of  drawings 
or  100  percent  of  3D  zones 

• 

• 

• 

• 

• 

6 

11 

9 

Test  an  early  system-level  integrated 
prototypes 

o 

• 

• 

• 

o 

1 

20 

5 

Establish  a  reliability  growth  curve 

• 

• 

• 

• 

• 

17 

6 

3 

Identify  key  product  characteristics 

• 

• 

• 

• 

• 

23 

0 

3 

Identify  critical  manufacturing  processes 

• 

• 

• 

• 

• 

21 

1 

4 

Conduct  producibility  assessments  to  identify 
manufacturing  risks  for  key  technologies 

• 

• 

• 

• 

• 

20 

3 

3 

Complete  failure  modes  and  effects  analysis 

• 

• 

• 

• 

• 

20 

2 

4 

/ 
,  <z> 


Other  26 
programs 

^o/— 


9  Practice  implemented 

o  Practice  not  implemented 

- Practice  not  applicable  or  information  not  available  per  the  program  office  response 


ACV  -  Amphibious  Combat  Vehicle 

AMDR  -  Air  and  Missile  Defense  Radar 

F-22  Inc.  3.2B  -  F-22  Increment  3.2B  Modernization 

JAGM  -  Joint  Air-to-Ground  Missile 

Space  Fence  Inc.  1  -  Space  Fence  Ground-Based  Radar  System  Increment  1 
Source:  GAO  analysis  of  DOD  data.  |  GAO-1 6-329SP 


•  Two  programs  have  not 
demonstrated  mature 
technologies,  yet  all  plan 
to  release  at  least  90 
percent  of  drawings. 

•  Two  programs  do  not 
plan  to  test  a  system- 
level  integrated 
prototype. 
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OBJ.  2  Obs.  3  -  Production  Start 


GAO 

Of  the  five  programs  that  held  a  production  decision  during  our  assessment  period, 
one  met  all  of  the  best  practices,  including  demonstrating  that  critical  processes  are 
in  statistical  control 


Knowledge-based  practices 
at  production  start 


4? 


£ 


¥ 


#■ 


9 


Demonstrate  all  critical  technologies  in  an 
operational  environment 

o 

• 

• 

• 

• 

5 

3 

3 

Release  at  least  90  percent  of  drawings 
or  100  percent  of  3D  zones 

• 

— 

• 

• 

• 

7 

2 

2 

Demonstrate  manufacturing  process 
capabilities  are  in  control 

o 

o 

o 

• 

o 

1 

7 

3 

Demonstrate  critical  processes  on  a  pilot 
production  line 

• 

• 

• 

• 

• 

6 

3 

2 

Test  a  production-representative  prototype 
in  its  intended  environment 

o 

• 

o 

• 

o 

4 

7 

0 

For  the  11 
non-shipbuilding 
programs  that 
have  reached 
this  juncture0 


^  Practice  implemented 

O  Practice  not  implemented 

- Practice  not  applicable  or  information  not  available  per  the  program  office  response 


•  SDB  II  met  all  of  the  best 
practices  for  production. 

•  Three  of  the  programs  do  not 
plan  to  test  a  production- 
representative  prototype 
before  making  a  production 
decision. 


pAB-T  -  Family  of  Advanced  Beyond  Line-of-Sight  Terminals 

JLTV  -  Joint  Light  Tactical  Vehicle 

MQ-4C  -  MQ-4C  Triton  Unmanned  Aircraft  System 

SDB  II  -  Small  Diameter  Bomb  Increment  II 

SSC  -  Ship  to  Shore  Connector  Amphibious  Craft 

Source:  GAO  analysis  of  DOD  data.  |  GAO-16-329SP 
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Objective  3:  Observations  on  current  and  future  programs 
implementation  of  acquisition  reforms  and  initiatives 

1 .  Of  the  55  current  and  future  programs  we  assessed,  37  have  established  an  affordability  constraint, 
similar  to  the  implementation  rate  in  our  last  assessment.  The  development  cost  growth  for  programs 
we  assessed  with  an  affordability  constraint  is  33  percentage  points  lower  than  the  median 
development  cost  growth  for  programs  without  a  constraint. 

2.  Of  the  43  current  programs  we  assessed,  39  have  conducted  a  “should-cost”  analysis  resulting  in 
anticipated  development  and  procurement  savings  of  over  $35  billion;  approximately  $21  billion  of 
these  savings  have  been  realized  to  date. 

3.  Of  the  55  current  and  future  programs  we  assessed,  43  plan  to  promote  competition  at  some  point 
during  acquisition.  Eight  of  the  43  current  programs  have  no  plans  for  competition  before  or  after 
development  start.  Half  of  the  future  programs  we  assessed  plan  to  conduct  competitive  prototyping. 

4.  Of  the  55  current  and  future  programs  we  assessed,  40  reported  software  development  as  a  high-risk 
area.  Programs  which  did  not  report  their  software  development  as  high-risk  have  experienced  greater 
schedule  delays.  Sixteen  of  the  43  current  programs  we  assessed  plan  to  concurrently  conduct 
production  and  software  development. 

5.  Sixteen  of  the  43  current  programs  we  assessed  are  in  production.  Eleven  of  these  programs  plan  to 
complete  30  percent  or  more  of  their  developmental  testing  after  production  start.  Further,  three  of 
these  11  programs  plans  to  place  more  than  20  percent  of  procurement  quantities  under  contract  before 
testing  is  complete.  For  the  programs  we  assessed,  as  concurrency  increases,  so  does  total 
acquisition  cost  growth. 
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OBJ.  3  Obs.  2  -  ACQUISITION  INITIATIVES 


GAO 

Of  the  43  current  programs  we  assessed,  39  have  conducted  a  “should-cost” 
analysis  resulting  in  anticipated  savings  of  over  $35  billion;  approximately  $21 
billion  of  these  savings  have  been  realized  to  date. 


Realized  should  cost  savings 

(in  fiscal  year  2016  dollars  billion) 


Went  to  outside  service  priorities 
Went  to  service  priorities 
Kept  in  program 
Destination  unknown 


Source:  GAO  analysis  of  survey  data.  |  GAO-16-329SP 


Of  the  39  programs,  35  identified  approximately  $35  billion 

in  realized  and  future  savings. 

•  $21 .2  billion  of  this  has  been  realized. 

•  Programs  reported  that  nearly  $286  million  of 
those  savings  were  used  to  offset  budget  cuts 
required  by  sequestration. 

•  Programs  may  not  have  strong  incentives  to  realize  or 
report  “should-cost”  savings  if  they  result  in  the  funding 
of  other  DOD  priorities. 

Programs  cited  several  activities  as  responsible  for  some  or 

all  of  their  “should-cost”  savings,  including: 

•  efficiencies  realized  through  contract  negotiations  (15 
programs), 

•  design  trades  to  balance  affordability  and  capability  (12 
programs),  and 

•  developmental  or  operational  testing  efficiencies  (7 
programs). 
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OBJ.  3  Obs.  3  -  ACQUISITION  INITIATIVES 


GAO 

Of  the  55  current  and  future  programs  we  assessed  43  have  acquisition  strategies  that  include 
some  strategies  for  encouraging  competition,  eight  will  not  conduct  a  competition,  and  4  are 
unsure  of  their  approach  at  this  time 


Current  Programs’  Plans  to  Promote  Competition  Future  Programs’  Plans  to  Promote  Competition 


Competition  before  system  development  start 

Competition  before  system  development  start 

22 

21 

6 

6 

Competition  after  system  development  start 


16 


Competition  after  system  development  start 


Competition  before  and  after  system  development  start 


\ 

\WB\NN 

8 

13 

13 

\ 

No 
Yes 
Neither 
Before  only 
After  only 
Both 

Don’t  know 


Source:  GAO  analysis  of  DOD  data.  |  GAO-16-329SP 


Competition  before  and  after  system  development  start 


No 

Yes 

Before  only 
After  only 
Both 


Don’t  know 

Source:  GAO  analysis  of  DOD  data.  |  GAO-16-329SP 
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Program  assessments 


There  are  currently  55  programs  that  will  be  covered  in  assessments 


•  42  2-page 

•  13  1-page 

1-page  assessments 


2-page  assessments 


APT 

DDG51  Flight  III 

F-15  EPAWSS 

IFPC  Inc  2-1  Block  1 

ITEP 

JSTARS  Recap 

LX(R) 

Ohio  Replacement 

P-8A  Inc  3 

PAR 

T-AO(X) 

UCLASS 

WSF 

3DELRR 

DDG  1000 

JLTV 

NGJ  Inc.  1 

ACV 

EELV 

JPALS  Inc  1A 

OASuW  Inc.  1 

AIM-9X  Blk  II 

EPS 

JTRS  HMS 

OCX 

AMDR 

F-22  Inc  3.2B 
Mod 

KC-46A 

PAC-3  MSE 

AMF  JTRS 

F-35  JSF 

LCS 

SDB  II 

AMPV 

FAB-T 

LCS  Packages 

Space  Fence 

Inc  1 

B-2  DMS-M 

G/ATOR 

LHA  6 

SSC 

CH-53K 

GPS  III 

M109A7  FOV 

VH-92A 

CIRCM 

IAMD 

MGUE  Inc  1 

WIN-T  Inc  2 

CRH 

JAGM 

MQ-4C  Triton 

WIN-T  Inc  3 

CVN  78 

MQ-8  Fire  Scout 
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Requirements  Really  Matter 


Travis  J.  Masters 

Assistant  Director 

Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 


For  more  information,  contact  Travis  J.  Masters,  masterst@gao.gov 
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Requirements  Knowledge  Is  Critical 


•  Positive  acquisition  outcomes  require  the  use  of  a  knowledge-based  approach  to  product 
development  that  demonstrates  high  levels  of  knowledge  before  significant  commitments 
are  made — knowledge  supplants  risk  over  time. 


•  An  executable  program  business  case  must  demonstrate  a  match  between  requirements 
and  available  resources  before  beginning  system  development. 


•  Keys  to  achieving  a  match: 

•  Clearly  defined  and  understood  requirements  -  down  to  at  least  a  completed  preliminary  design 

•  Resource  knowledge  -  available  technologies,  time  ,  money,  and  skilled  people 


Page  25 


GAO 


What  Are  Requirements? 


•  It  depends,  because  “where  you  sit  determines  where  you  stand” 

•  Operational  requirements 

•  Typically  rooted  in  capability  gaps 

•  Establish  capability  needs  and  set  operational  goals 

•  Provide  the  high-level  success  criteria 

•  Design  requirements 

•  Also  known  as  derived  requirements  or  specifications 

•  Identify  the  key  subsystems  (technologies)  and  interfaces  through 
preliminary  design  and  ultimately  full  (or  critical)  design 

•  Provide  a  clear  understanding  of  technical  /  design  feasibility  of  a  system 
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Requirements  Creep  or  Knowledge  Gained? 


•  For  Major  Defense  Acquisition  Programs,  operational  requirements  like  Key 
Performance  Parameters,  don’t  tend  to  change  after  development  begins, 
but  as  operational  requirements  are  decomposed  the  number  of  design 
requirements  (i.e.  specifications)  grows  significantly,  often  giving  the 
appearance  of  “creep.” 

•  Growth  in  the  number  of  design  requirements  is  a  natural  process  for 
every  system  as  requirements  are  decomposed  and  the  system  design 
matures. 


1^  MilesIWKi 

o  r --r: i .“R— t  al.'l  hmH  mlpjlrni. 


Source:  GAO-15-469 


SOuiM  ■GwG-  DClD  fii  tv  *>3  | 
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Characteristics  of  More  Successful  Programs 

•  Incremental,  Derivative,  and/or  Prototypes 

•  Limited  system  complexity  and  high  technology  maturity 

•  Requirements  informed  through  systems  engineering 

•  Cost  and  schedule  estimates  informed  through  systems  engineering 

•  Examples  include: 

•  KC-46A 

•  Joint  Light  Tactical  Vehicle 

•  Small  Diameter  Bomb 

•  P-8A 
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Typical  DOD  Requirements  Flow 


ICD 

I 

CDD 


Performance  Specification 

Typical  DOD  Program  Start  - -| - 

Functional  Baseline 


i 

Allocated 

1 

Baseline 

Dost  ■  raciicc  i  rogram  wiai t 

ri 

1 1 

1  1 

1 1 

i  1 

1 

Product  Baseline 
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Systems  Engineering  Builds  Knowledge 

•  Implementation  of  the  SE  processes  begins  with  the  identification  of  a 
validated  operational  need. 

•  The  technical  processes  ...  include  the  top-down  design  processes  and 
bottom-up  realization  processes  that  support  transformation  of  operational 
needs  into  operational  capabilities  (i.e.  a  delivered  weapon  system). 


•  During  the  Technology  Development  (TD)  phase: 

•  Program  Manager  and  Systems  Engineer  work  to  reduce  technical  risk 
and  develop  a  sufficient  understanding  of  the  materiel  solution 
development 

*  Validate  design  approaches  and  cost  estimates,  to  refine 
requirements  and  to  ensure  affordability  is  designed  in  to  the  desired 
capability... (i.e.  the  top-down  design  processes  cited  above) 


Source:  Defense  Acquisition  Guidebook 
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The  Requirements  Part  of  Systems  Engineering 


Requirements 


Systems  Engineering 


Typical  DOD 
program  start 


Product 
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Considerations  For  Further  Improving  Weapon 
System  Acquisition  Performance 


•  Cultural  shift  to  get  rid  of  the  “we  can’t  do  that  here”  attitude 

•  DOD  5000  is  flexible  and  statute  does  not  prohibit  it 

•  DOD  programs  have  done  it  successfully 

•  Think  incremental  or  derivative — build  on  what  is  known 

•  Consider  knowledge  as  an  independent  variable — properly  align  decisions  with 
knowledge,  don’t  try  to  force  knowledge  to  the  left  of  a  schedule-driven  process 

•  Think  outside  the  box  about  ways  to  work  with  the  industrial  base  to  better 
understand  design  requirements  before  program  start 

•  Different  contracting  types 

•  Competition  through  PDR  or  even  CDR 

•  Require  a  “prototype  or  derivative  solution” 

•  Consider  ways  to  work  with  the  Congress  to  identify  ways  to  invest  in  systems 
engineering  knowledge  —  pay  me  now  or  pay  me  later 
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Acquisition  Reform 


Efforts  to  Streamline  DOD’s  Acquisition 
Process  and  Reduce  Burden  on 
Commercial  Companies 


For  more  information,  contact  Cheryl  Andrew  at  andrewc@qao.gov 
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•  Recent  Reform  and  Improvement  Efforts 

•  Impact  of  DOD’s  Implementation  of  Reform  Efforts  on  Internal 
Processes 

•  Efforts  to  Streamline  DOD’s  Oversight  of  Acquisition  Programs 

•  Future  Congressional  Areas  of  Emphasis 
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Recent  Reform  and  Improvement  Efforts 


People 


DAWIA 


Faster, 
Better,  ~~ 
Cheaper 


Acquisition  Reform  of  the 
90’s 


Defense  Reform  Initiative 


Knowledge  &_ 
Savings 


WSARA 


Rebuild  the  acquisition  workforce 

Adopt  commercial  practices  and  standards 

Reduce  infrastructure  and  overhead 
costs  and  improve  business  process 

W  Improve  acquisition  program  cost 
*  and  schedule  outcomes 


Improve  DOD’s  purchasing  power 


1990’s 


2015 


5/9/2016 
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Impact  of  DOD’s  Implementation  of  Reform  Efforts  on 
Internal  Processes 


Average  time  to  complete  information  requirement  (months) 

24  High  value  Moderate  value  Less  than 

moderate 

value 


Source:  GAO  analysis  of  DOD  data.  |  GAO-15-192 


Review  levels-  traditional  program 


Office  of  the  Secretary  of  Defense 
Milestone  Decision  Authority 


Functional  Offices  Senior  Managers 


Functional  Offices  Staff 


Service  Acquisition  Executive 


Functional  Offices  Senior  Managers 


Functional  Offices  Staff 


Program  Executive  Officer 


Program  Executive  Officer  Staff 


Review  levels-  some  classified  programs 


Board  of  Directors  •  Milestone  Decision  Authority 

Defense  Acquisition  .  .  . 

r  Service  Secretary 

Executive 


Service  Chief  of  Staff 


Service  Acquisition 
Executive 


Specific,  focused  interactions  with  a  small 
number  of  functional  offices  as  necessary 


Program  Executive  Officer 


Program  Office 

1 

1 

Program  Office 

Source:  GAO  presentation  of  DOD  data.  |  GAO-15-192 


Page  37 


GAO _ 

Impact  of  DOD’s  Implementation  of  Reform  Efforts 
on  Internal  Processes 


Organizations  Typically  Involved  in  the  Review  Process  for  an  Air  Force  Acquisition 
Strategy 


30  OSD 
organizations 


Office  of  the  Secretary  of  Defense  (OSD)  Level 


•  Defense  Acquisition  Executive 

•  Vice  Chairman  of  the  Joint  Chiefs  of  Staff 

•  Under  Secretary  of  Defense  (Policy) 

•  Under  Secretary  of  Defense  (Comptroller) 

•  Under  Secretary  of  Defense  (Personnel  &  Readiness) 

•  Under  Secretary  of  Defense  (Intelligence) 

•  Chief  Information  Officer 

•  Director,  Operational  Test  &  Evaluation 

•  Director.  Cost  Assessment  and  Program  Evaluation 

•  Director,  Acquisition  Resources  &  Analysis 

•  Principal  Deputy  Under  Secretary  of  Defense  (Acquisition,  Technology.  &  Logistics) 

•  Assistant  Secretary  of  Defense  (Acquisition) 

•  Assistant  Secretary  of  Defense  (Logistics  &  Material  Readiness) 

•  Deputy  Under  Secretary  of  Defense  (Installations  and  Environment) 

•  Deputy  General  Counsel  (Acquisition  &  Logistics) 


Assistant  Secretary  of  Defense  (Research  &  Engineering) 

Deputy  Assistant  Secretary  of  Defense,  Strategic  &  Tactical  Systems 
Deputy  Assistant  Secretary  of  Defense.  Space  &  Intelligence 

Deputy  Assistant  Secretary  of  Defense,  Communication,  Command,  and  Control  Cyber 

Director.  National  Geospatial-Intelligence  Agency 

Deputy  Director  Cost  Assessment 

Director,  Defense  Pncmg 

Director,  Systems  Engineering 

Director,  Developmental  Test  &  Evaluation 

Deputy  Assistant  Secretary  of  Defense.  Manufacturing  &  Industrial  Base  Policy 

Director.  International  Cooperation 

Director,  Performance  Assessment  and  Root  Cause  Analysis 

Assistant  Secretary  of  Defense  (Legislative  Affairs) 

Director,  Defense  Procurement  and  Acquisition  Policy 

Assistant  Secretary  of  Defense  (Operational  Energy  Plans  and  Programs) 


22  Service-level 
organizations 


Service  Acquisition  Executive  (SAE)  Office  Level 


•  Assistant  Secretary  of  the  Air  Force  for  Acquisition  (Service  Acquisition  Executive) 

•  Assistant  Secretary  of  the  Air  Force  Installations  &  Environment 

•  Air  Force  Logistics.  Installations.  &  Mission  Support 

•  Air  Force  Operations,  Plans,  &  Requirements 

•Air  Force  Intelligence,  Surveillance,  &  Reconnaissance 

•  Air  Force  Financial  Management  &  Comptroller 

•  Air  Force  Test  &  Evaluation 

•  Assistant  Secretary  of  the  Air  Force  Small  Business  Programs 

•  Assistant  Secretary  of  the  Air  Force  Chief  Information  Officer 

•  Assistant  Secretary  of  the  Air  Force  Test  &  Evaluation  (Policy  and  Programs) 

•  Air  Force  Operations,  Plans,  &  Requirements  (Operational  Capability  Requirements) 

•  Air  Force  Logistics.  Installations  &  Mission  Support  (Logistics) 


Assistant  Secretary  of  the  Air  Force  Installations  &  Environment  (Logistics) 

Air  Force  Intelligence  Surveillance.  &  Reconnaissance  (Strategy.  Plans.  Doctrine  & 
Force  Development) 

Assistant  Secretary  of  the  Air  Force  Chief  Information  Officer  (Policy  &  Resources) 
Assistant  Secretary  of  the  Air  Force  Deputy  General  Counsel  for  Acquisition 
Air  Force  Financial  Management  and  Comptroller  Deputy  Assistant  Secretary  (Cost 
and  Economics) 

Air  Force  Financial  Management  and  Comptroller  Deputy  Assistant  Secretary  (Budget) 
Assistant  Secretary  of  the  Air  Force  Directorate  of  Science,  Technology  &  Engmeenng 
Assistant  Secretary  of  the  Air  Force  Directorate  Management  Policy  &  Program  Integration 
1  Assistant  Secretary  of  the  Air  Force  Directorate  of  Contracting 
Air  Force  Acquisition  Capability  Directorate 


4  PEO-level 
organizations 


Program  Executive  Office 


•  Program  Executive  Officer  •  Functional  Staff:  contracting,  logistics,  finance,  and  engineering 

•  Deputy  Program  Executive  Officer  •  Program  Executive  Officer  Execution  Group 

Source:  GAO  presentation  of  DOD  data  |  GAO-15-192 
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Impact  of  DOD’s  Implementation  of  Reform  Efforts  on 
Internal  Processes 


Material  Solution 
Analysis 

Technology 

Development 

Engineering  and 
Manufacturing 
DeveloDment 

^ /f\ /r\ 


Challenge  is  Determining  the  Right  Balance  of  Oversight  and  Insight 
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Efforts  to  Streamline  DOD’s  Oversight  of  Acquisition 
Programs 


FY16NDAA 


Milestone  Decision  Authority  for  most  programs  will  now  be  the  service  acquisition  executive  instead  of  OSD 

Milestone  Decision  Authority  makes  a  determination  about  an  acquisition  program’s  “business  case”  at 
Milestone  A  and  a  certification  at  Milestone  B 

OSD  Systems  Engineering  and  Developmental  Test  and  Evaluation  Offices  will  serve  in  an  advisory  role  and  are 
not  directly  in  the  chain  of  command  for  approving  programs 

Some  required  documentation  was  eliminated  -  such  as  stand  alone  manpower  estimates 


DOD  Improvement  Efforts 


OSD  and  the  Navy  are  pilot  testing  a  streamlined  reporting  process  with  the  Next  Generation  Jammer  program 
OSD  delegated  24  programs  to  the  services  for  oversight  in  FY  15 

OSD  and  the  services  are  using  an  electronic  coordination  tool  to  review  and  approve  acquisition  strategies 

DAU  is  analyzing  data  it  collected  on  5  MDAPs  related  to  the  number  of  higher  level  reviews  and  the  timeline  for 
approval  of  key  acquisition  documents 


5/9/2016 
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Future  Congressional  Areas  of  Emphasis 


•  Obtaining  Technologically  Advanced  Weapon  Systems  Quicker 
(Acquisition  Agility) 

•  Prototyping 

•  Open  Systems 

•  Reducing  Burden  on  Commercial  Companies  with  Changes  to: 

•  Commercial  Item  Determinations 

•  Other  Transaction  Authority 

•  Technical  Data  Rights 

•  Cost  and  Pricing  Data 
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GAO  Review  on  Reducing  Burden  on  Commercial 
Companies 


•  GAO  is  currently  conducting  a  review  to  identify  impediments 
innovative  companies  perceive  or  face  when  conducting 

business  with  POD 

•  We  are  conducting  12-15  case  studies  to  illustrate  the 
impediments  companies  face.  The  majority  of  the 
companies  are  involved  with  cyber  security,  data  analytics, 
or  robotics. 

•  We  plan  to  analyze  DOD  policies  and  initiatives  aimed  at 
eliminating  impediments 
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GAO  Review  on  Reducing  Burden  on  Commercial 
Companies 


•  Most  common  concerns: 

•  Lengthy  decision-making  process  (commercial  item  and 
price  reasonableness  determinations) 

•  Multiple,  costly  software  certifications 

•  Unstable  funding  and  budget  inflexibility 

•  Intellectual  property  rights 

•  Requirements  that  are  too  prescriptive 

•  Difficulty  identifying  business  opportunities 

•  Acquisition  workforce  lacks  subject-matter  and  technical 
expertise 

•  Risk  averse  workforce 

•  Rotating  DOD  workforce 


Page  43 


GAO 


GAO  on  the  Web 

Web  site:  http://www.qao.gov/ 

Congressional  Relations 

Katherine  Siggerud,  Managing  Director,  sigqerudk@qao.gov 
(202)  512-4400,  U.S.  Government  Accountability  Office 
441  G  Street,  NW,  Room  7125,  Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  vounqc1@qao.gov 
(202)  512-4800,  U.S.  Government  Accountability  Office 
441  G  Street,  NW,  Room  7149,  Washington,  DC  20548 

Copyright 
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The  published  product  may  be  reproduced  and  distributed  in  its  entirety  without  further  permission 
from  GAO.  However,  because  this  work  may  contain  copyrighted  images  or  other  material, 
permission  from  the  copyright  holder  may  be  necessary  if  you  wish  to  reproduce  this  material 
separately. 
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